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Abstract 



This paper studies diffusion processes constrained to the positive orthant, and in- 
vestigates changes in the steady-state distribution of such diffusions under infinitesimal 
changes in the drift. Our first main result states that any constrained function and its 
drift-derivative is the unique solution to an augmented Skorohod problem. Our second 
I""! main results uses this characterization to prove that the steady-state distribution of the 

Pm joint processes (diffusion and its derivative processes) satisfies a basic adjoint relation. 

Ph We specialize the technique to the case of reflected Brownian motion. 

-(— > 

2 1 Introduction 

This paper is motivated by a desire to better understand the relation between performance 

^"^ metrics and control variables in a network with shared but Hmited resources. We are specif- 

I ically interested in service networks, where customers seeking a certain service may suffer 

t~^ from delays as a result of temporary insufficient service capacity. The control variables are 

'^ the service capacities at the individual stations. Many service processes can be modeled 

by stochastic (or queueing) networks, and an important question is how resources should 
^-^ be allocated given random fluctuations in the arrivals and its interplay with potentially 

^—1 random service times. Questions of this type are readily answered if the network has a 



product-form structure Kleinrock (1964); Wein (1989), but few results have been obtained 



when this assumption fails Dieker et al. (2011 ); Pollett (2009). It is the goal of this paper to 



k>< introduce new tools in this context, which could be used in the context of both sensitivity 

5^ analysis and system optimization. 

We study diffusion processes and their 'derivatives', defined as the change in the process 
under an infinitesimal change in the drift. Although some of our results are stated more 
generally, this paper focuses on diffusion processes for two reasons. First, this framework 
allows us to explain key concepts in a tractable yet relatively general setting. Second, dif- 
fusion processes are rooted in heavy-traffic approximations for stochastic networks, and the 
heavy-traffic assumption seems reasonable in the context of resource allocation problems 
with systems operating close to their capacity. Another feature of this paper is that we 



focus on the long-term (steady-state) behavior of diffusfons and their derivatives. Although 
it is certainly desirable to obtain time-dependent tools as well, given the vast body of work 
on steady-state results, making this assumption is a natural first step. The techniques 
developed in this paper are likely to be also relevant in the time-dependent case. 

We have two main results. The first is a statement on the behavior of deterministic 
functions under the well-known Skorohod reflection map with oblique reflection (regula- 
tion), and states that the map and its 'derivative' are the unique solution to an augmented 
version of the Skorohod problem. Our proof of this result relies on recent insights into di- 



rectional derivatives by Mandelbaum and Ramanan (2010), which have been developed in 



the context of time-inhomogeneous systems but are shown here to be useful for sensitivity 
analysis as well. 

For our second main result, we specialize to diffusion processes and study the steady- 
state distribution of the diffusion process jointly with its derivative processes. Specifically, 
given a constrained diffusion process Z representing the dynamics of the underlying stochas- 
tic network (i.e., the queue lengths at each of the stations), we let the stochastic process 
A represent the change in Z under an infinitesimal change in the drift. This process has 
jumps even if Z is continuous. Our result is that the steady-state distribution of the joint 
processes (Z, A) satisfies a kind of basic adjoint relation, which is the analog of the equation 
vr'Q = for continuous-time Markov processes on a discrete state space. The proof relies 
on a delicate analysis of the jumps of A. 

The intuition behind the program carried out in this paper can be summarized as 
follows. Suppose Z^ is a constrained diffusion process with drift coefficient /x(-) — ev in the 
interior of the orthant, where v is an arbitrary nonnegative vector. Suppose the processes 
{Z*^} are driven by the same Brownian motion for every e > 0, so that they are coupled. 
The processes Z = Z^ and Z'' are Markovian, and one can therefore expect to be able to 
give a basic adjoint relationship for their stationary distributions (if they exist). Moreover, 
(Z, Z*^) and therefore (Z, (Z*^ — Z)/e) can be expected to be Markovian as a result of the 
coupling. Provided one can make sense of the limit as e — )• 0+, one can expect that the 
distribution of the limit satisfies a similar relationship. This results in an 'augmented' basic 
adjoint relationship, which we state in Theorem [3| The constrained diffusion processes 
studied in this paper are reflected Brownian motions (RBMs). Using standard tools, our 
results can be formulated for the much larger class of constrained diffusion processes such 
as pathwise solutions to stochastic differential equations with reflection [Dupuis and Ishn 



(1991); Ramanan (2006), but we focus on RBM to be able to highlight the main ideas. 

When carrying out the aforementioned approach for reflected Brownian motion, we were 
surprised to find that the natural analog of the one-dimensional basic adjoint relationship 
(or the product-form solution with normal reflection) fails to characterize the stationary 
distribution of {Z,A). In fact, even though Z is known not to spend any time on low- 
dimensional faces, it is critical to incorporate the jumps of A when Z reaches those faces 
in order to formulate the basic adjoint relationship. 

This work has the potential to lead to new numerical methods in the context of opti- 



mization and sensitivity analysis for queueing networks, which remove the need for com- 
putationally intensive or numerically unstable operations such as gradient estimation. To 
explain, due to the division by e, any performance metric of {Z^ — Z'')/e suffers from 
numerical instability issues for small e > 0. Researchers in stochastic optimization have 



developed several techniques to mitigate this effect (see, e.g., Asmussen and Glynn (2007)). 
The approach taken in this paper is to analytically describe and investigate the dynamics 
of the limit. Even for networks not in heavy traffic, this could lead to new bias reduction 
methods in the context of stochastic optimization. 

The framework of this paper is related to a body of literature known as infinitesimal per- 



turbation analysis Glasserman (1991, 1993, 1994); Heidergott (2006). Infinite perturbation 



analysis also aims to perform sensitivity analysis or gradient estimation for performance 
metrics in (say) a queueing network, and it does so by formulating conditions under which 
an expectation and a derivative operator can be interchanged. Here, howevever, we do not 
seek such an interchange involving a performance metric but instead we study the (whole) 
steady-state distribution of a stochastic process with its derivative process. 

This paper is outlined as follows. Section [2] summarizes our approach in the one- 
dimensional case, which serves as a guide for our multi-dimensional results. Section [3] 
discusses two technical preliminaries: oblique refiection maps and their derivatives. In 
Section |4] we formulate our two main results. Section [5] is devoted to the proof of the first 
main result, while Section [6] gives the proof of the second main result. A key role is played 
by jump measures, for which we obtain a description in Section [7| The appendices contain 
several technical digressions. 

Notation 

For J € N, M denotes the J-dimensional Euclidean space. We denote the space of real 
n X m matrices by M"^™, and the subset of nonnegative matrices by M"^™. All vectors 
are to be interpreted as column vectors, and we write M^ and Mj for the j-th column and 
the i-th row of a matrix M, respectively. In particular, Vi is the i-th element of a vector 
V and Mv' is element (i, j) of a matrix M. Similarly, given a set / C {1, . . . , J}, we write 
Mj and M for the matrices consisting of the rows and columns of M, respectively, with 
indices in /. Throughout, E stands for the identity matrix and we write S^ for E^. We use 
the symbol ' for transpose. 

Given a space S, a vector-valued function h : S ^ M"^ on S, and a vector of measures 
v = {ui, . . . uj) on S, we set 



Mx)K^x) = /M.)..(d.) 



provided the right-hand side exists. We shall also employ this notation when h is matrix- 
valued. That is, we write for h : S ^ M''^'^, 



/ h{x)v{dx) = I {h{x),v{dx))i 



where (•, •)hs is the Hilbert-Schmidt inner product on M''^'^ given by 

(Mi,M2)Hs = tr(M(M2). 

For a function g : M"^^"^ — )• M, we define Vg : M"'^"' — )• M"^^"^ as the function for which 
element {i,j) is given by the directional derivative of g in the direction of the matrix 
with only zero entries except for element (i,j), where its entry is 1. We also write, for 
i = 1,...,J, Fi = {{z, a) eRix M-^^-^ : Zi = 0}, Ff = {{z, a) e R-[ x M'^^'' : a^ = 0}. 
The space of functions / : M.'^ x M._^_^ — )• M which are twice continuously differentiable 
with bounded derivatives is denoted by C^O^i x M:[^-^). 

We write O'^ for the space of M:j^-valued functions on M4. which are right-continuous 
on M+ with left limits in (0, 00). The subset of continuous functions is written as C"^, and 
Cf_ denotes the set of nonnegative continuous functions. Similarly, we write n -^ for the 
space of M"'^ •^-valued right-continuous functions on M+ with left limits. The subset of 
M-l^-^ -valued functions is denoted by B-l^-^. 

2 A motivating one-dimensional result 

Let Z he a one-dimensional reflected Brownian motion with drift 6 < and variance o"^. 
That is, 

Z{t) = X{t)+Y{t)>0, 

where X{t) is Brownian motion with drift 6 and variance cr^, and the regulating term Y is 
given by 



Y{t) = max sup [-X{s)],0\ . 

\0<s<t J 

Similarly, for e > 0, let Z'' be one-dimensional Brownian motion with drift 9 — e and 
variance a^. It follows from Mandelbaum and Massey (1995) that 



A{t)= lim ^{Z{t)-Z'{t)) (2.1) 

exists as e — )• 0+. Moreover, we have an explicit formula: 

A{t)=t-B{t), (2.2) 

where 

B{t) = sup{s e [0, t] : Z{s) = 0}, 



and sup = by convention. In view of the definition of ^ in (2.1 ), we call it the derivative 
process of Z. 




t 

Figure 1: Sample paths of {Z,A) as a function of time. The sohd black curve is Z, while 
the dashed red curve is A. The slope of ^ is 1 whenever it is continuous, and A jumps to 
whenever Z hits 0. 



We now relate these notions to sensitivity analysis. Making the dependence on 9 ex- 
plicit, our investigations are motivated by the following sequence of equalities: for any 
'smooth' function (performance measure) (j), 



d9 



E 



•^(^'(oo)) 



E 






HZ\oo)) 



E 



A%oo)(P'{Z%oo)) 



(2.3) 



Thus, to study (infinitesimal) changes in the steady-state performance measure under in- 
finitesimal changes in the drift 9, one is led to investigating the stationary distribution 
of {Z,A) (assuming it exists). We are indeed able to justify the above equalities in the 
one-dimensional case. 

One readily checks that the sample paths of the process B are nondecreasing, that they 
are left-continuous with right-hand limits, and that A has jumps. In particular, the process 
A is of finite variation and (Z, A) is a semimartingale with jumps. An illustration of the 
process (Z, A) is given in Figure [I] From Ito's formula in conjunction with sample path 
properties of ^, we obtain the following result. We suppress further details of the proof, 
since this program is carried out in greater generality in the proof of Theorem [3j 

Theorem 1. Let Z he a one- dimensional reflected Brownian motion with drift 9 and 
variance a"^. Let A he defined in (2.2). Suppose that the process {Z,A) has a unique 
stationary distrihution n. For any f € C^(M+ x M+), we have the following relationship: 



2^ 

2" 0^2 



a-^:r^f{z,a) + 



'^/(^,a) + ^/(^,a) 



d_ 
da 



fiO,a) 



■K(dz, da) 



d_ 
dz 



f [0,0)9. 



(2.4) 



One can go further and derive the Laplace transform of vr using this theorem, see 
Appendix [A} One then finds that, for any a,r] > 0, 

<2fi 

e-^'-v^Tridz, da) = (2.5) 

aa^-e + V2r?cj2 + 02 

In particular, the theorem completely determines the stationary measure it. It is also 
possible to derive this result immediately from standard fluctuation identities for Brownian 



motion with drift, using results from D§bicki et al. (2007). In fact, since the corresponding 



densities are known explicitly (or can be found by inverting the Laplace transform), it is 
possible to write down the density of (Z(oo),^(oo)) in closed form. Using the resulting 



expression, it can be verified directly that (2.3) indeed holds. 



3 Oblique reflection maps and their directional derivatives 

This section contains the technical preliminaries to formulate a higher-dimensional analog 
of Theorem [T] To introduce the analogs of the derivatives a and b, we need the following 
definition. 

Definition 1. (Oblique reflection map) Suppose a given J x J real matrix R can be written 
as R = I — P, where P is a nonnegative matrix with spectral radius less than one. Then 
for every x G D"^, there exists a unique pair {y,z) € Ii'l_ x W^ satisfying the following 
conditions: 

1. z{t) = x{t) + Ry{t) >0 fort>0, 

2. y(0) = 0, y is componentwise nondecreasing and 

z{t)dy{t) = 0. 



/o 
We write y = ^{x) and z = T{x) for the oblique reflection map. 

The reflection map gives rise to derivatives as formalized in the following definition. 



Existence of the derivatives is guaranteed by Theorem 1.1 in Mandelbaum and Ramanan 



(2010). 



Definition 2. (Derivatives of the reflection map) Let xit) = tE and define the M - 
valued functions a and b by setting a = lim^^Q^a^ and b = linie^o+^e! where, for j = 
I,..., J, 



al 



^ ^[r(x)-r(x-6x^-)], (3.1) 



6^ = __ [$(2;) - <I)(x - ex-')] . 



e 
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Then we have for each t > 0, 

a{t) = tE-Rb{t). (3.2) 

For notational convenience, we write a = T'{x) and b = —^'(x). 

4 Main results 

This section states the main results of this paper. The first result makes the connection 
between derivatives and an augmented Skorohod problem, which we define momentarily. 
The second result is a basic adjoint relationship for the steady-state distribution of reflected 
Brownian motion and its derivative process, which is the analog of the equation tt'Q = 
mentioned in the introduction. 

4.1 Augmented Skorohod problems and derivatives 

In this section we introduce the augmented Skorohod problem and connect it with deriva- 
tives of the oblique reflection map. 

Definition 3. (Augmented Skorohod problem) Suppose we are given two JxJ real matrices 
R = I — P and R = I — P, where both P and P are nonnegative matrices with spectral 
radius less than one. Given {x, x) £ C"^ x C'^ with x nonnegative and nondecreasing, we 
say that {z, y, a, b) G C:l x C:f x D^^ x D_,_^ satisfies the augmented Skorohod problem 
associated with {R, R) for {x,x) if the following conditions are satisfied: 

1. z{t) = x{t) + Ry{t) for t > 0, 

2. y(0) = 0, y is componentwise nondecreasing and 



oo 



z{t)dy{t) = 0. 







3. a{t) = x{t) - Rb{t) for t > 0, 

4- b{0) = 0, b{t) > 0, b is componentwise nondecreasing and, for j = 1, . . . ,J, 

oo 

z{t)dh>{t) = 0. (4.1) 



5. For i = 1, . . . , J and t >0, Zi{t) = implies ai{t) = 0. 



Building on results from Mandelbaum and Ramanan (2010), we show in Appendix B 



that the augmented Skorohod problem has a unique solution. To interpret solutions to the 
augmented Skorohod problem, we found it easiest to think of the dynamics of {z,a^) for 

7 



\ 





t = 

pushing 
irection 



t = 1 



Ah 



t = 2 



t = t^l- 

o 



X 



■■*,.= 1 t = 2 t = 3 



A\ 



Aj 



t 


= 1- 








c 


3 




f = 3 


t = 








X 


t-^ 1 




i = 2 



^! 



Figure 2: The first diagram depicts a trajectory of Z, with corresponding 'free' path X 
(dotted). In the second and third diagram, the trajectories of A^ and A^ travel at unit 
rate right and up, respectively, until Z hits c?M^. The face Z2 = is hit at time t = 1, 
causing A^ and A^ to jump to the faces ^2 = ^'^d A| = 0, respectively, in direction R^. 



each j = 1, . . . , J separately. When z hits the face z/ = 0, then a^ jumps to the face o]- = 
in the direction of the unique vector in the column space of R which brings it to that face. 
We refer to Figure [2] for an illustrative example in the two-dimensional case. 

Unlike requirements 2 and 4 in Definition [Sj requirement 5 is not a 'complementary' 
condition. In view of the sample path dynamics in Figure [2| it may seem reasonable to 
replace requirement 5 by L a^{t)dy{t) = or another complementarity condition between 
(y, z) and (a, h). In that case, however, the augmented Skorohod will fail to have a unique 
solution. This can be seen by verifying that both the left-derivative and the right-derivative 
of the reflection map satisfy /g a^{t)dy{t) = but only the left-derivative (as defined in 
Definitional satisfies requirement 5. 

We now make a connection between derivatives (sensitivity analysis) and solutions to 
the augmented Skorohod problem. 

Theorem 2. Fix some x € C , and let z = T{x) and y = $(a:) he given by the oblique 
reflection map. Define the derivatives a = r'(x) and b = — ^'(a;) as in Definition^ Set 
x{t) = tE for t > 0. Then {z,y,a,b) satisfies the augmented Skorohod problem associated 
with {R, R) for (x, x)- 

4.2 Steady-state distribution of RBM and its derivative process 

Our second main result shows that the stationary distribution of reflected Brownian motion 
jointly with its derivative process satisfies an extension of the basic adjoint relationship 
(BAR) for refiected Brownian motion. The proof relies on Ito's formula in conjunction 
with properties developed in the previous section, and the result can be formulated for a 
much larger class of constrained diffusion processes with continuous sample paths such as 
strong solutions to SDERs. To be able to focus on the critical ideas, we only present the 
proof for reflected Brownian motion. 

We start with some definitions. When X is a J-dimensional Brownian motion with 
X{0) = X with drift 9 and covariance matrix T,, Z = T{X) is a so-called reflected Brownian 
motion. The process Y with sample paths Y = ^(X) is called the regulator of X. We 
note that it is not always possible to deflne reflected Brownian motion through a pathwise 



mapping, see for instance Dieker (2010) for details and references. Throughout this paper, 
we only work with reflected Brownian motions obtainable through the oblique reflection 
map of Definition [T| but extensions of our result are presumably possible. 

We define the following operators: for any / G C^(]R:| x M"^^"^) and each set I C 
{1,2,..., J}, let 

iOif){z,a) = Y. i-lf^ f{nsuiz,Qiia)), (4.2) 

sc{i,...,j}\/ 

O = Y. Oi, (4.3) 

7C{1,...,J} 



where Usui is the projection operator which sets the coordinates in S U I equal to 0, and 
Qj is also an projection operator with the following property. The matrix Qi{a) is obtained 
from a by subtracting columns of R^ , in such a way that the rows of Qi{a) with indices in 
I become zero. Explicitly, we have 



Qi{a) = a-R\R\)-^ai, 



(4.4) 



where R\ is the principal submatrix of R obtained by removing rows and columns from R 
which do not lie in /. When / = 0, we set Qi{a) = a for a G M . 



We also define operators L and T on C^iJ 



itJxJ 



) as follows: 



1 ^ 



'' .E./ 



dzjdzi 



j=i 



dzi 



(4.5) 



where we recall that, in view of our notational conventions, tr(Va) is shorthand for 

Z^i,j=l daij • 

We can now formulate the following theorem, which is our second main result. 

Theorem 3. Let X be a Brownian motion with drift 9 and covariance matrix S. Let 
Z = T{X) be the resulting reflected Brownian motion and A = T'{X) be the derivative of Z 
as in Definition^ Suppose {Z, A) has a unique stationary distribution it. Then there exists 
finite Borel measure u such that for any f G C^{M.'^ x M_|_^ ), the following relationship 
holds: 



I 



[ToOf]{z,a)dTr{z,a)+ [R'V^{Of){z,a)]du{z,a) = 0, (4.6) 



is given in (6.6) below. 



where the operators O and T are given in {4-3) and {4-5), and the Laplace transform of v 



We remark that the proof of the theorem shows that (4.6) is equivalent to several 

< M;^'"'^) and each set / C {1, 2, . . . , J}, vr must 



equations. That is, for any / G (7^(1 
satisfy 



/ 



[ToOif]{z,a)dTr{z,a) + 



UAP^i^F-) 



[R'V,{Oif)iz,a)]di^{z,a) = 0, (4.7) 



where the operators 0/ are defined in (4.2). Note that (4.7) produces 2 equations, one of 
which is trivial. 



It is natural to ask whether (4.6) or (4.7) fully determines the stationary distribution vr. 



It is outside the scope of this paper to attempt to prove this, but we give some supporting 
evidence by showing that: 1) restricting to functions only depending on z and not on a 
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yields the classical BAR, 2) in the one-dimensional case, we recover the relationship from 
Section^ 3) if i? = (normal reflection directions), vr factorizes as a product of J measures 
('product-form'), 4) varying R leads to a different equation. 



We first check that (4.6) yields the classical BAR for the stationary distribution of the 
reflected Brownnian motion Z when choosing f(z,a) = g{z). One readily checks that in 
this case. 



{Of){z,a) 



Y, E (-i)"''5(n5u7^) 

/C{1,2,...,J}SC{1,...,J}\/ 



9i.z)-g{Q). 



Substituting the above equation in (4.6), we immediately obtain the well-known basic 



adjoint relationship Harrison and Williams (1987a): 



Lg{z)dTT{z) + / [R'V,g{z)]di^{z) 



(4.8) 



where dii^z) = /^gj^jx j dvr(2:, a) is the stationary distribution for Z and the Borel measure 
du{z) is given by dv{z) = J^^j^jxj dv{z, a). 

We next check that (4.6) recovers Theorem [II which shows in particular that (4.6) fully 
determines vr in the one- dimensional case, i.e., for J = 1. Indeed, it is readily seen that 

(0/)(z, a) = (O^Diz, a) + (0{i}/)(z, a) = f{z, a) - /(O, a). 



Combining this with (4.6) gives (2.4) 



Finally, we verify that (4.6) determines vr for J > 2 if i? is identity matrix, i.e., P = 0. 
In this case, we have by definition that Aij = for i ^ j- To determine the joint stationary 
distribution of {Z, A), which is of product form, it suffices to show that (4.6 ) or (4.7) yields 
the correct marginal distributions n^dzj, dajj 
If we set f{z, a 



for each J = 1, 2, . . . , J. Let /i G (7^(1 



h{zj, ajj) and / = {1, 2, . . . , J}\{j} in ( |4.7[ ), we deduce that 
{Oif){z,a) = h{zi,ajj) - h{0,ajj). 



Substituting this into (4.7), and using the fact that total mass of u is —9 when Z is of one 



dimension, we immediately get the one-dimensional result (2.4), which characterizes the 



marginal distribution TT(dzj,dajj). Therefore, (4.6) also determines vr in this case. 

We next give an informal argument why (4.7) for / = cannot be sufficient to de- 



termine TT, even though the equation involves an integral of vr over the interior and all 
low-dimensional faces and this equation determines vr completely in the one-dimensional 
case. Theorem [3] gives the basic adjoint relationship for the augmented Skorohod problem 
associated with {R, R) for {X, x), but a similar equa tion can be given with {R, R) replaced 
by {R,R). This would lead to the same equations (4.7), but the definition of Q[ in (4.4) 
would involve the matrix R instead of R. For 1 = 0, however, this equation beco mes 
independent of R. One expects vr to depend on the choice of R, in which case (4.7) for 
1 = cannot determine vr. 
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5 Characteristics of derivatives and proof of Theorem [2] 

In this section, we prove Theorem [2} We also collect additional sample path properties of 
derivatives, with an emphasis on their jump behavior. These properties will be used in the 
proof of Theorem [3J 

Throughout this section, we work under the conditions of Theorem [2j That is, we 
assume that x G C"^ is given and we write z = r(x), y = $(a;), a = T'{x) and b = — $'(x). 
We also set x{t) =tE iovt>0. 

5.1 Complementarity 

This section connects the augmented Skorohod problem associated with (R, R) for (X, x) 
with {z,a). Note that, in view of Definitions [I] and ^ the first two requirements of the 
augmented Skorohod problem in Definition [3] are immediately satisfied for {x,y,z). It is 
immediate that a = x ~ R^ by definition of a, so we must indeed choose R = R. We 
proceed with showing that a and b lie in D_,_^ as required for the augmented Skorohod 
problem, but it is convenient to first establish part of the fourth requirement. 

Lemma 1. The Wl -valued function b is componentwise nonnegative and nondecreasing. 

Proof. Since x{t) = ^-^ for ^ ^ 0, x is evidently nonnegative and nondecreasing. The 



monotonicity result in Theorem 6 of Kella and Whitt (1996) shows that for any fixed 



e > 0, each component of b^ is nonnegative and nondecreasing. The lemma follows from 
the fact that b is the pointwise limit of the sequences {b^} as e — t- 0+. □ 



JxJ „.r,u,„^ A,:v,^-f-;^^<, ^ „^^ u u„ i^ TnJxJ 



Lemma 2. The Wl -valued functions a and b lie in 

Proof. Since b is nonnegative in view of Lemma [T| we have shown the claim for b after 
verifying that b G n . We deduce from Theorem 1.1 in Mandelbaum and Ramanan 



(2010) that each component of b is upper semicontinuous and that it has left and right 
limits everywhere. Since b is nondecreasing by Lemma [l| these properties imply that 

be o-^^-^. 

We next show that a £ D , ^ . Clearly, since b £ D , ^ , we only need to show that a is 



nonnegative. Again by the monotonicity result in Theorem 6 of Kella and Whitt (1996), 
for any fixed e > 0, each component of a^ is nonnegative. This completes the proof of the 
lemma after letting e — )• 0+. □ 

We next investigate the fourth and fifth requirement of Definition [Sj To this end, we 



need a characterization of b which relies heavily on Mandelbaum and Ramanan (2010). 



Lemma 3. b is the unique solution to the following system of equations: for i,j = 1, . . . ,J, 
and t > 0, 

b>{t)= sup [6ls + [PVUs)], 
se*(i)(<) 
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where the supremum over an empty set should be interpreted as zero and 

<I>(,)(t) = {sG[0,t]:zi{s) = 0}. (5.1) 



Proof. We use Theorem 1.1 of Mandelbaum and Ramanan (2010), which can be simphfied 



in view of Lemma [T] and the nonnegativity of the matrix P. This theorem states that 

l^(t) = i' if*e(0,t(,)), 

' \sup,e*„(t)['^^ + [^V],(s)] if i G [t(,),oo), 

where t(i) = inf{t > : Zi{t) = 0} and ^(i)(t) = {s G [0,t] : Zi{s) = Q,Yi{s) = Yi{t)]. 
Observe that, again using Lemma [T| the supremum must be attained at the rightmost end 
of the closed interval ^ u\{t). Since Y is nondecreasing, this is also the rightmost point of 
the closed set ^u\{t). This establishes the lemma in view of the convention used for the 
supremum of an empty set. □ 

Lemma 4. Fix any j = 1, . . . ,J, we have 

oo 

z{t)dV{t) = 0. (5.3) 



Proof. Fix some i = 1, . . . , J. Note that if Zi{t) > at time t, we deduce from the path 
continuity of z that there exists some e > such that Zi{s) > for s G (t — e,t + e). This 
implies that <^(i)(s) is constant as a set- valued function for s G (t — e, t + e). Thus 6j(s) is 



constant for s G {t — e,t + e) by (5.2). Since i is arbitrary, this yields (5.3). □ 

Lemma 5. If Zi{t) = for some i, then we have ai{t) = 0. 

Proof. Suppose Zi{t) = 0. In view of Lemma [T| we deduce from (5.2) that, for any j = 
1,...,J, 

m = 5it+[p'vut). 



Now it follows from ( [3^ and R = I - P' that 

a^{t) = 6lt- [WUt) = 6it -^i{i) + [P'VUt) = 0, 
which completes the proof of the lemma. □ 

The above two lemmas together with Lemma [2] yield two further complementary con- 
ditions. 

Corollary 1. For any j = 1, . . . , J, we have 

oo 

a\t)dy{t) = 0, (5.4) 

o 

a\t)dh>{t) = 0. 
/o 

Proof of Theorem^ The claim is now immediate from (3.1 ) in conjunction with Lemmasfll 



millandlS} □ 
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5.2 Jumps of a 

In this section, we collect sample path properties of a related to its jump behavior. This 
plays a critical role in the derivation of Theorem |3j 

The next lemma states that a is linear whenever z is in the interior of M:{_. 

Lemma 6. If z{t) G M:[\3M:[ for t G [a,/3], then we have for t G [a,/3] 

a{t) = a{Q) + {t — a)I. 
In particular, a is continuous on {a, 13) and can only have jumps when z G &&:\. 



Proof. In view of (3.2), it suffices to show that h is constant for t G [a,/?]. Since z{t) G 
M:j^\5M:j^ for t G [a,/3], we obtain from (5.1) that for each i = 1, . . . , J, ^u\[f) is constant 



as a set- valued function. Therefore, we deduce from (5.2) that 6(t) is a constant in 



[fjx J 



for t G [a, /?]• The proof of the lemma is complete. □ 

For any function g on M+, we write Ag{t) = g{t) — g{t—). In view of the above lemma, 
we can characterize the continuous part of the function a. Formally, we write 

a{t) = a''{t) + a'^{t), 

where 



a"' 



\t) = Y,^a{s). 

s<t 

We have the following corollary. 

Corollary 2. a%t) = a'=(0) + tE for any t > 0. 

We next characterize the jump direction of a when a jump occurs. 

Lemma 7. Fix a nonempty set I C {1, 2, . . . , J} and some t > 0. Suppose that Zk{t) = 
for k £ I and Zi{t) > for i ^ I. If Aa{t) ^ 0, then we must have 



Aa{t) = -Y,R''[^bUt). 



kei 



Proof. Since Zi{t) > for i ^ I, we deduce from the sample path continuity of z that there 
exists some e > such that for i ^ I, Zi{s) > for s G {t — e,t]. This yields that for i ^ I, 



$(j)(s) is a constant as a set-valued function for s G {t — e,t]. From (5.2) we infer that 
for i ^ I, bi{s) is constant for s G (i — e,t]. This implies that [A6]j(i) = for i ^ I, and 
therefore that 

J 
Aa{t) = -RAb{t) = -Y,R'^[AbUt) = - ^i?'^[A6]fc(t). 

fc=l k&I 

This completes the proof of the lemma. □ 
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6 A basic adjoint relationship and proof of Theorem [3] 

This section is devoted to the proof of Theorem [3} The key idea is to apply Ito's formula 
to the semimartingale {Z, A) and use sample path properties of {Z, A) to analyze the 
stationary measure. Although this is a standard approach in the context of reflected 
Brownian motion, the analysis here is delicate due to the presence of jumps in the process 
A. 



6.1 Ito's formula for the semimartingale {Z,A) 

In this section, we apply Ito's formula to the semimartingale {Z, A). 

We first show that {Z, A) is a semimartingale, i.e, each of its component is a semi- 
martingale. Loosely speaking, a semimartingale is an adapted process which is the sum of 
a local martingale and a finite variation process, with sample paths in D. For more details. 



we refer readers to (Protter, 2005 Ch. 3) or (Jacod and Shiryaev 2003, Ch. 1). 

Lemma 8. {Z, A) is a semimartingale. 

Proof. Clearly, {Z,A) is a process adapted to the natural filtration of X. We know from 
Lemma [2] that each component of the process {Z, A) lies in D. Since Z is a semimartingale, 
to show (Z, A) is a semimartingale, it suffices to show that ^ is a semimartingale. In fact, 
from Lemma IT] and (3.2) we immediately deduce that A is a finite variation process, that 
is, the paths of A are almost surely of finite variation on [0, T] for any T > 0. In particular, 
^ is a semimartingale. □ 

Since Z has drift 6 and covariance matrix S, we can rewrite Z as follows: for each 
t>0, 

Z{t) = Z(0) + et + W{t) + RY{t), 

where W{t) is a driftless Brownian motion with covariance matrix S. 

By Ito's formula, e.g., ( | Jacod and Shiryaev| |2003| Sec. 1.4), for any / G C^^^Rf xM^^''), 
we have 

f{Z{t),A{t)) = fiZiO),A{0))+ f VJiZ{.s),A{.s-))dWi.s) 



+ [ [R'VJ{Z{s),Ais-))]dYis) 
Jo 



+ [ Lf{Z{s),A{s-))ds+ I VafiZis),Ais-))dA%s) 
Jo Jo 

+ Y^[f(Z{s),Ai.s)) - f{Zis),Ai.s-))]. 

s<t 



(6.1) 



(Note that, compared to the formulation in Theorem 1.4.57 of Jacod and Shiryaev (2003), 
we have absorbed the last sum of the jump part into the integral Jq Vaf{Z{s), A{s—))dA^{s) 
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Suppose that, under appropriate conditions, {Z,A) is positive recurrent and has a 
unique stationary distribution vr. After taking expectation with respect to vr on both 



sides of (6.1), the second term involving dW on the right-hand side vanishes since it is a 
martingale term. By Corollary [2] and a standard argument, we have 

K^f Vaf{Z{s),A{s-))dA''{s) =t f tv{Vaf{z,a))d7riz,a). 



We conclude that, for each i > and each f & C^{ 



'2/-Tn>J 



rrJx J\ 



= tf[Tf{z,a)]d7T{z,a) + E^f[R'VJ{Z{s),A{s-))]dY{s) 
+E^ ^ [f{Z{s),Ais)) - f{Zis),Ais-))], 



(6.2) 



s<t 



where T is given in (4.5). This equation serves as the starting point for proving Theorem 3 



6.2 The boundary term 



In this section we rewrite the boundary term in (6.2), i.e., the term involving dY. Let 
u = [ui, . . . , vj) be the unique vector of measures on dM.'^ x M'^^'^ for which 

/ h{z,a)u{dz,da) = E^ I h{Z{s),A{s-))dY{s), 



for all continuous h : dM.'^ x M — )• M with compact support. Note that the right-hand 
side is always finite as a consequence of the fact that E^^Y^l) < oo, see (Harrison and 



Wilhams, 1987b, Section 



Our next goal is to give a characterization of measure u in terms of vr, which we carry 
out through Laplace transforms. We start with determining the support of v. 

Lemma 9. The support of i' is IJi(-^« ^ ^i)- 

Proof. In view of Lemma [2j it is clear that A can have at most countably many jumps. 



For any continuous h : dl 



rrJx J 



with compact support, we have 



h{Z{s),A{s-))dY{s)= I h{Z{s),A{s))dY{s), 

JO 

since the measure dY is continuous and the integrand has countably many jumps by 
Lemma [H It follows from the definition of v that 



/i(z, a)v{dz, da) = E^ / h{Z{s),A{s))dY{s). 





(6.3) 



The complementarity conditions L Z{t)dY{t) = and (5.4) imply the lemma. 



□ 
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On combining Equations (6.2) and (6.3) we obtain that for any / G C^^K'^ x ivii_|_ ;, 

[R'VJ{z,a)]du{z,a) 

(6.4) 



[Tf{z, a)]dTT{z, a) + 
+ ^-E^ Y, [f{Z{s), A{s)) - f{Z{s),A{s-))]. 



t 



s<t 



We now express the Laplace transform of z^ in terms of the Laplace transform of vr. Set 
fiz,a) = exp{-r]-z-a-a) £ C^(Mf xM^''"') where {r],a) G M:[ xM^'"'^. After substituting 
/ in (6.4), we obtain 



Q{r],a)TT*{ri,a) -^{R''n)jiy*{'q,a) + H{r],a) = 0, 



(6.5) 



where 



-r]'T,r] — O'rj — tr(a), 



^^—•q-z—a-a 



d7r{z,a), 



Q{v,a) 
TT*{r],a) 

ilR:[xM-'X"' 

i,*{r],a) = [ e~'^-'-''-''diyj{z,a), 

JF.nFf 

■' J 



S<1 



Dividing (6.5) by rjj > and letting rjj — )• oo, we deduce that 



u* {rj,a) = - Ejj Jirn^ rjjTr* (r/, a) , 



r)j—¥oo 



(6.6) 



where we have used the fact that Vj{Fj n F,j) = for i 7^ j so that lim^^._!.oo '^j*(^> a) = 
by the dominated convergence theorem. Under further regularity conditions on vr, one 
can use the initial value theorem for Laplace transforms to show that dvj = ^^jjdiTj for 
an appropriate restriction ttj of vr. Carrying out this procedure provides little additional 
insight, and we therefore suppress further details. 

6.3 The jump term 



We now proceed investigating the jump term, i.e., the term in (6.2 ) involving the countable 



sum. It is possible to express this term completely in terms of tt using the theory of 
distributions; this is done in Section [7} However, to prove Theorem [3j it is possible to use 
(6.2) for special functions with the property that the jump term vanishes. 
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Throughout, we fix a set / C {!,..., J}. Recall the definition of O/ in (4.2). It 
is our aim to show that the jump term vanishes for functions of the form O//, where 
/ G C^lM."^ X M'^^'^) as before. We first introduce a lemma. 

Lemma 10. For any f : M:[ x M'^^-' — ;> M, if Zj = for j ^ /, then for any a G M"'^'^ we 
have 

Yl {-lf^f{Usuiz,a) = 0. 
sc{i,...,j}\i 

In particular, if Zj = for j ^ /, then we have Ojf{z,a) = 0. 

Proof. Suppose Zj = for some j ^ /. Then for any set SC {1, . . . , J}\/ with j ^ S, we 
have Hsuiz = Ilsuiu{j}Z- Using this observation, we deduce that 

^ (-l)l^l/(n5u/^,a) 
5C{1,...,J}\/ 

E (-l)l''l/(n5u/^,a)+ Y. {-lf^msuiz,a) 

jesc{i,..,j}\/ j(^sc{i,...,j}\i 

E (-i)i^i/(n5u/^,a)+ Yl (-l)""/(n5u/ub-}^,«) 
jesc{i,...,j}\i j(^sc{i,...,j}\i 

Y i-lf ^f{nsuiz,a)+ Y (-l)'^'"V(n5u/^,a) 

ie5c{i,...,j}\/ iesc{i,...,j}\i 

= 0. 

The proof of the lemma is complete. □ 

Now we are ready to show that the jump term vanishes for special functions O//. For 
any K C {1, . . . , J}, Zk denotes the process whose components are those of Z with indices 
in if. 

Lemma 11. For each t>0 and f G C^(]R:[ x M"^^-^), we have 

E^ Y [Oif{Z{s),A[s)) - Oif{Z{s),A{s-))] = 0. (6.7) 

s<t 

Proof. By Lemma [6] and Lemma [Toj we have 
E^ Y [OifiZis),Ais)) - OifiZ{s),Ais-))] 

= Y ^- H [Oif{Z{s),Ais))-Ojf{Z{s),Ais-))] 

0^A-C{1,...,J} s<i:Zjf(s)=0,Z{i,...,j}\jf{s)>0 

= Y ^^ E [OjfiZis),Ais))-OjfiZis),Ais-))]. 
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Therefore, to show (|6.7|) it suffices to show for each nonempty set K <^ I, we have 



E. Y, [Oif{Z{s),A{s))-Oif{Z{s),A{s-))]=0. 

s<t:ZA'(s)=0,Z{i,..,,j}\j^{s)>0 



(6i 



To prove (6.8) we first deduce from Lemma [T] that when Zj^{s) = and ■^|i,...,j}\_ft:(s) > 0, 

Qk{A{s-) = A{s)). 
Next, since i^ C /, we use the projection property of the operator Qj to obtain 

QiiA{s)) = Qi{Qk{A{s-)) = Qi{A{s-)). 



Now (6.8) readily follows from and the definition of 0/ as in (4.2). Thus we have completed 
the proof of the lemma. □ 



Proof of Theorem^ Equation (4.7) immediately follows from (6.4) and Lemma 11 Sum- 
ming all the equations in (4.7) over the sets / C {1, . . . , J}, we obtain (4.6). □ 



7 Jump measures 



In this section, we further investigate the jump term in (6.2 ), resulting in a characterization 



of jump measures in terms of the stationary distribution vr. We first introduce some 



notation. For / C {1, . . . ,J},I ^ 0, we define measures uj on 



,\n 



■JxJ 



with support 



in(0,oo)l^'l 



ir JxJ 



. We set, for Borel sets B C (0, oo)l^'l, C C 



If^ 



ui{B,C) = E^ Y, ^c{A{s-)}. 

s<l:Zi{s)=0,Zic{s)eB,A{s)=/=A{s-) 



We write zj for the subvector of z consisting of the components with indices in / as before, 
and we also let z\j denote the projection oi z to {z : zjc = 0}. We start with an auxiliary 
result on the measures uj. 

Lemma 12. For each I C {1, . . . , J}, / 7^ and k = 1, . . . , J, we have ^/({(z/c, a) : at = 
0}) = 0. 

Proof. We exploit the dynamics of the augmented Skorohod problem. Since ^^(s— ) = 
implies Zk{s) = 0, we have ui{{{zi<^,a) : au = 0}) = for /c G I'^. We next consider k £ I. 
Since the continuous part of A^ is strictly increasing when Z^ > 0, the only possibility for 
Zj{s) = 0, Ak{s—) = 0, and A{s) / A{s—) to occur simultaneously is for Z to hit the face 
zj = without leaving the face Zk = 0. Since the time Z spends on the boundary has 
Lebesgue measure zero, this cannot happen almost surely. □ 
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To proceed with our description of the measures uj, we need tools from theory of 
distributions (or generahzed functions), which makes it possible to differentiate probability 



distributions. For background on this theory, see Duistermaat and Kolk (2010); Rudin 



( 1991 ). For / C {1, . . . , J}, we define the operator 



T? 



1 






^11 



d' 



ijdi ^ jei 



E' 



d 



^^'^1 nCT J 



tr(Va). 



Note that, with the understanding that we use the theory of distributions, this operator 
can act on probability measures. We also define 

d'Ki{zic^a)= I dTT{z,a). 

J Zj 

The main result of this section is that uj can be expressed in terms of it. Indeed, together 



with Lemma 12, it completely determines uj. 



Proposition 1. For each / C {1, . . . , J}, / / 0, we have, with zjc G {0,oo)^^''\a G M'^^'^ 
and Ofc / for k = 1, . . . , J, 

duj{zjc,a)= Y. (-1)''^'''/ [T*Kcd7rK]{zK^,a). 



KCI,Kf!^d 



Zl\K 



Proof. The basis for the proof is ( |6.4[ ). We first rewrite the jump term in (6.4) using the 
jump measures. In view of Lemmas [5] and [7j 



S<1 



Y, E^ E [f{Z\K4s),A{s))-fiZ\Ms),A{s-))] 

II}f^KC{l,...,.J} s<1:Zk{s)=0,Zkc(s)>0 

E ^- E [f{Z\K4s),QK{Ais-)))-fiZ\K^{s),Ais-))] 

9¥=KC{1,...,J} s<l:ZK{s)=0,ZKcis)>0,A(s)y^A{s~) 



E / [f(.z\K'=,QK(.a)) - f{z\K'=,a)]duK{zK'=,a) 



Fix some nonempty / C { 1 , . . . , J} . For f £ C^ 



'2{it$J V i\mJ^J') 



with the property that / 



vanishes on Uje/c ^i U Ui ^t^ (6.4) thus reduces to 

Tf{z,a)dTr{z,a) = ^ / fiz\i<^uL,a)duj\L{zicuL,a). 



I^LCI 
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If moreover f(z, a) does not depend on z/, this can be simplified further: 

/ Tf{z,a)dTT{z,a) = V] / f{z\ic,a) dui\L{zia^L,a). (7.1) 

The left-hand side can be rewritten using the theory of differentiation for distributions 



(Duistermaat and Kolk, 2010 Ch. 4) or (Rudin, 1991 Sec. II.6.12). This leads to 



Tf{z,a)dTr{z,a) = / f{z\ic,a)[TjcdiTi]{zic,a). 



Combining this with (7.1) and rearranging terms, we get 
/ f{z\ic,a)dui{z 

J zic.a 



[zic,a 



f{z\jc,a)[Tjcd7ri]{zic,a) - 



^ / f{z\ic,a) / duj\L{zicyjL,a). 

^j^^(k t^t' ^I''<^ •' ^L 

This shows that, for z/c E (0, 00)^^"^, a G M::^^'^ and a^ / for /c = 1, . . . , J, 
dui{zic,a) = Tjcd7ri{zjc,a) - ^ / duj\i{zi<^yjL,a). 

Since \I\L\ < \I\, this representation allows us to finish the proof of the proposition by 
an elementary induction argument on |/|. Alternatively, one could use a version of the 
inclusion-exclusion principle (Stanley 1997, Sec. 2.1). □ 
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A Proof of (2.5) 



This appendix uses Theorem [T] to find the Laplace transform of the stationary distribution 
vr of (Z, A) in the one-dimensional case, thereby showing in particular that Theorem 
completely determines tt. Writing C{a, rj) for the Laplace transform of tt, Theorem 
implies that 

-CT^a^ -aO-rA C{a, r/) + 7]C{0, rj) + aO = 0. (A.l 
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In particular, on setting r/ = ^a^a^ — a9 we get 



(T^a^ - ad 



C [ 0, -a^a^ - a9 ] + aO = 0. 



After substitution of a = (6* + ^/9^~+~2(Th]) ja^ , we find that 



r]C{0,r]) 



G^ 



Laplace transform given in (2.5). 



Substituting this back into (A.l) and simplifying the resulting expression, we obtain the 



B The augmented Skorohod problem and uniqueness 

In this appendix, we prove that the augmented Skorohod problem admits a unique solution. 



To this end, we employ a similar contraction map as in Lemma 3.6 of Mandelbaum and 
Ramanan (2010 1 . Define a map A from D-^^-^ to D-^^-^ by setting, for t > 0, 



mm 



sup \^\{s) + \P\P\{s)\. 



(B.l) 



Momentarily we show that A is a contraction map, and thus A has a unique fixed point h. 
This also implies that given h^^' = 0, and recursively define 6("'+^) = K{h^^'), we have for 
every T > 0, ||6^"^ — 6||t — ^ as n — )• oo. Since x is nonnegative and nondecreasing and P 
is nonnegative, we deduce that 6^"^ is componentwisely nonnegative and nondecreasing for 
each n. Therefore, we obtain that the fixed point b is also nonnegative and nondecreasing. 
Now let a = X ~ R^-, ^ = r(a;), and y = <I>(x). We now verify directly that {z,y,a,b) is 
a solution to the augmented Skorohod problem. Only the fourth and fifth requirement in 
Definition [3] are not immediate. The fourth requirement can be shown to hold using the 
same argument as in the proof of Lemma |4| For the fifth requirement, we note that if 
Zi{t) = 0, (B.l) implies that for each j, 



which yields 



m = xm + ipmt)^ 



ai{t) = Xtit) - {Rb)i{t) = Xi{t) + (PbUt) - h{t) = 0. 



To establish the uniqueness of solutions to the augmented Skorohod problem, we use 
the contraction map A. Suppose (z, y, a, b) solves the augmented Skorohod problem. Let 
b = A(6). If we can show that b = b, meaning 6 is a fixed point of A, then it follows from 
the uniqueness of the fixed point that there must be a unique solution to the augmented 
Skorohod problem. Suppose there exists some i,j and to such that 6^ (to) ¥" H{to)- We 



22 



discu ss tw o cases. If Zi^to) = 0, using nonnegativity and monotonicity of b, one can check 



from (B.l) that ^^(io) = Xiito) + [Pb^]i{'to)- From the definition of the augmented Skorohod 
problem, we also know that Zi{tQ) = implies a^(to) = xK^o) + [Rb^hito) = 0. Therefore, 
we have 6^ (to) = ^(^o)) a contradiction. Now consider the second case where we have 
2^4(^0) > 0- If the set $(j)(to) is empty, we have 6^ (to) = ^K^o) = ^i^) = 0. If not, let s 
be the maximal element in <i>(j)(to). We deduce from the previous case in conjunction with 



the complementarity condition (4.1) that 6^(to) = bl{s) = b^is) = ^^(to)- This is again a 
contradiction. Therefore, we obtain b = b and infer that the augmented Skorohod problem 
has a unique solution. 



It remains to show that A is a contraction map on D , which is equipped with 



the uniform norm on compact sets. As in the proof of Lemma 3.6 in Mandelbaum and 



Ramanan 



(2010) we assume that, without loss of generality, the maximum row sum of P 



is r/ < 1. It is easy to verify that for any fixed T > 0, 

\\A{b)-A{b')\\T<v\\b-b'\\T 

for all b, b' G D*^^"^. Thus we have proved the existence and uniqueness of a fixed point for 
A. 
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